Field of the Invention
The present invention relates to a data processing apparatus, a data processing method and a storage medium for temporarily storing image data before and after image processing.
Description of the Related Art
Conventionally, local (neighborhood) image processing such as spatial filtering is performed when forming an image and outputting the formed image. In the local image processing, a predetermined calculation is performed on a spatial filtering region including pixels to be processed (hereinafter simply referred to as “processing pixels”) by using all or most of the pixels in the spatial filtering region.
Resolution conversion is one type of local (neighborhood) image processing, in which an input image is converted to an image of desired size by enlarging or reducing the input image. In the resolution conversion, naturally, the number of input pixels and the number of output pixels after processing are significantly different. Accordingly, it is difficult to implement an apparatus for resolution conversion as compared to an apparatus for common local (neighborhood) image processing. For example, in Japanese Patent Laid-Open No. 2010-087804, it is pointed out that in a case where a plurality of combinations of delay memory and spatial filtering are used, if resolution conversion is executed between each combination, two delay memories before and after the processing cannot be used effectively.
To address this problem, Japanese Patent Laid-Open No. 2010-087804 discloses resolution conversion processing called “main scanning scaling” in which the number of pixels in the main scanning direction of an input image is changed by size enlargement or reduction. According to the technique disclosed in Japanese Patent Laid-Open No. 2010-087804, two spatial filters before and after resolution conversion perform calculation on the processing pixels in a two-dimensional region, and thus it is necessary to provide a delay memory for each of the two spatial filters. Also, Japanese Patent Laid-Open No. 2010-087804 focuses on the two delay memories and changes in the number of pixels, and discloses a technique for making effective use of the memories. Also, Japanese Patent Laid-Open No. 2010-087804 uses block (tile) processing in which an image is two dimensionally divided into regions in both the main scanning direction and the sub scanning direction, and image processing is performed sequentially on pixel regions obtained as a result of the division.
Meanwhile, Japanese Patent Laid-Open No. 2006-139606 discloses a band processing technique that is similar to the block (tile) processing described above. With the band processing, an image is one-dimensionally divided into regions only in the main scanning direction, and pixel regions obtained as a result of division is sequentially subjected to image processing. With this technique, in order to perform local (neighborhood) image processing on the band regions without a break, the band regions are configured such that adjacent band regions partially overlap each other at their boundaries. Also, according to Japanese Patent Laid-Open No. 2006-139606, pixels are scanned on a pixel by pixel basis in the same direction as the height of each band region, and thereby the capacities of the delay memories for holding processing pixels required for local (neighborhood) image processing are defined by the magnitude of the height of each band region. In this way, Japanese Patent Laid-Open No. 2006-139606 implements reduction of the capacities of the delay memories.
As with the block (tile) processing disclosed in Japanese Patent Laid-Open No. 2010-087804, the band processing disclosed in Japanese Patent Laid-Open No. 2006-139606 is also a technique in which an image is divided into regions and image processing is performed sequentially or in a distributed concurrent manner. Hereinafter, such techniques will be collectively referred to as “region division techniques”. Image processing that is incompatible with such region division techniques is not only the resolution conversion disclosed in Japanese Patent Laid-Open No. 2010-087804.
For example, trimming processing, edge extension processing and the like that are used in printer image processing or the like are also incompatible with the region division techniques. Trimming processing is processing for partial removal of an image, and the number of pixels of an output image is reduced as compared to the number of pixels of an input image. Likewise, edge extension is processing performed only on the pixels on the edge of an image, in which only a predetermined number of pixels are copied and extended or the pixels of a specific color are extended, and the number of pixels of an output image is increased as compared to the number of pixels of an input image.
IP conversion and the like that are used in video processing or the like are also incompatible with the region division techniques. IP conversion is processing in which interlace images whose pixels have been thinned out in the sub scanning direction are combined into a progressive image in which the pixels of one frame are not thinned out by using a plurality of frames in the time axis direction. For example, in the case where a progressive image of a single image is obtained by combining interlace images of two frames, the total number of pixels of an input image is the same as the number of pixels of an output image. However, with high quality IP conversion, a progressive image of a single frame is obtained by combining interlace images of 3 to 5 frames, and thus the number of pixels of an output image is reduced as compared to that of an input image. In the example of IP conversion, the relationship between the number of pixels of an input image and the number of pixels of an output image varies depending on the operation mode (image quality mode) selected by the user.
Recently available image processing apparatuses incorporate many various image processing tasks including the above described spatial filtering, resolution conversion, trimming processing, edge extension processing and IP conversion. In order to achieve the desired image quality with each apparatus, the image processing apparatus often selects processing from among the various processing tasks according to the application and uses the selected processing tasks in combination.
However, additional attention is required in the case where a combination of a plurality of processing tasks is implemented with a region division technique. In the case where calculation is performed using processing pixels of a two-dimensional region such as spatial filtering, it is necessary to accumulate surrounding pixels according to the pixel size handled by the filtering and process the image. If spatial filtering is performed a plurality of times, it is necessary to accumulate surrounding pixels each time and process a larger image in which all of the surrounding pixels have been accumulated.
Furthermore, with spatial filtering such as chroma upsampling in video processing, only accumulating surrounding pixels as described above is insufficient. With such spatial filtering, it is necessary to give consideration to the phase of input processing pixels on an image. Accordingly, there is a special limitation that restricts the numbers of pixels in the vertical and horizontal directions of the image input for the spatial filtering to multiples of 4.
In order to perform the desired image processing while satisfying the limitations of a plurality of processing tasks selected and combined according to the application, it is not possible to easily define the relationship between the number of pixels of the input image and the number of pixels of the output image. The application is designed to process an image having a number of pixels that is greater than that of an image obtained as a result of accumulation of surrounding pixels in spatial filtering performed a plurality of times. In some cases, the application is required to remove a part of the processed image at the end of processing so as to adjust the image to the desired size.
When using the region division technique, it is important to consider how to implement the image processing method that selects and combines a plurality of image processing tasks by using hardware.
Generally, an image processing apparatus that uses hardware implements the simultaneous operation of various components and the cooperative operation of the components by transferring images to and from an external storage device via an input local buffer and an output local buffer. However, as described above, when performing image processing by selecting and combining a plurality of processing tasks, the number of pixels of input image and the number of pixels of output image cannot be easily defined. Accordingly, it is difficult to appropriately define the capacity of the input local buffer and the capacity of the output local buffer, and a problem arises such as reduction of utilization efficiency caused by preparing excessively large buffers or delay of processing by preparing excessively small storage capacity.
Japanese Patent Laid-Open No. 2010-087804 focuses on two (pre-filter and post-filter) delay memories before and after main scanning scaling. However, Japanese Patent Laid-Open No. 2010-087804 does not give consideration to an image processing apparatus as a hardware configuration for implementing an image processing method for selecting and combining a plurality of various image processing tasks. Accordingly, Japanese Patent Laid-Open No. 2010-087804 contains no disclosure of a useful solution for a method that copes with the above-described situation in which the number of pixels of the input image and the number of pixels of the output image vary. Also, Japanese Patent Laid-Open No. 2010-087804 contains no disclosure of a useful solution that focuses on the input local buffer that is required to read images from an external storage device and the output local buffer that is required to write processed images into the external storage device.
The present invention has been made in view of the above problems and provides a technique that enables efficient utilization of a storage device that temporarily stores an input image and an output image under the conditions in which the number of input pixels and the number of output pixels vary due to image processing in which various processing tasks are combined.